Cross-Account AWS S3 Bucket Transfer Guide
Overview
This guide provides instructions for transferring data between S3 buckets that are in different AWS accounts.
Prerequisites
- AWS CLI installed and configured
- Source bucket:
open-rvfcast - Destination bucket:
rvfcast - IAM user in source account:
open-rvfcast-data
Setup for Cross-Account Transfer
1. Configure Destination Bucket Policy
Apply this policy to the destination bucket (rvfcast):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::SOURCE_ACCOUNT_ID:user/SOURCE_USER_NAME"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::DESTINATION_BUCKET_NAME",
"arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
]
}
]
}
2. Update Source Account IAM User Policy
Apply this policy to the IAM user in the source account:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME",
"arn:aws:s3:::DESTINATION_BUCKET_NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObjectTagging"
],
"Resource": "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
}
]
}
Performing the Sync
Option 1: Sync Using Profile with Source Bucket Access
If you have a profile with access to the source bucket:
-
Configure your AWS CLI profile:
aws configure --profile source-profile -
Run the sync command:
aws s3 sync s3://SOURCE_BUCKET_NAME s3://DESTINATION_BUCKET_NAME --size-only --profile source-profile
Option 2: Sync Using Profile with Destination Bucket Access
If you have a profile with access to the destination bucket:
-
Configure your AWS CLI profile:
aws configure --profile destination-profile -
Configure source account credentials:
aws configure --profile source-profile -
Run the sync command with source and destination profiles:
# If your AWS CLI version supports it: aws s3 sync s3://SOURCE_BUCKET_NAME s3://DESTINATION_BUCKET_NAME --size-only --profile destination-profile --source-profile source-profile # If not supported, you'll need to use Option 1 instead
Additional Flags
--size-only: Compare file sizes only (faster than comparing checksums)--no-progress: Hide progress information--dryrun: Show what would be transferred without actually transferring
Troubleshooting
If you encounter "Access Denied" errors:
- Verify both bucket policy and IAM user policy are correctly configured
- Check for organization-level policies that might restrict cross-account access
- Ensure the Block Public Access settings aren't interfering with your policies
- Verify that Object Ownership settings aren't blocking your access
The most common issue is missing necessary permissions in either the bucket policy or the IAM user policy.